Crawler- based search engines are considered the most familiar to the people mainly because the companies are able to develop their own software which helps in enabling to maintain and build searchable databases of the web pages. It also helps in organizing those pages into the most valuable and most applicable way to the users. The term crawler is used by the companies as the software that is produced tends to crawl the web just like a spider which helps in automatically add and update the new pages to the search index which helps the user and makes their work easier. If we look into the history of search crawlers, it can be said that the first crawler was the World Wide Web Wander which had developed in the year 1993 (Baker, 2005).